AD  738157 


HumRRO  RBP-D4- 70-1 


\ 

HumRRO 


•  oi.  y 

E5531 


! 

if 


F 


Approved  for  public  release; 

distribution  unlimited 


((•produced  by 

NATIONAL  TECHNICAL 
INFORMATION  SERVICE 

Springfield,  Vr.  221S1 


!  ?r"1  a  v 


r-’.-i 

i  i 

J  i  ;~L 


I.  i. 


C  ' 


HUMAN  RESOURCES  RESEARCH  ORGANIZATION 


*! 


Best 

Available 

Copy 


rnf+*<rf* 


JfRAPHIC  DATA  1.  Report  No.  2. 

HumRRO  RBP-D4-70-1 

3.  Recipient’s  Accession  No. 

•  and  .Subtitle  < 

3UIDL  FOR  DEVELOPING  QUESTIONNAIRE  ITEMS 

5.  Report  Date 

Jan  70 

6. 

J»or(») 

0.  Jacobs 

8.  Porlorming  Organization  Kept. 

No.  HumRRO  RBP-D4-70-1 

rforming  Organization  Nume  and  Address 

jnan  Resources  Research  Organization  (HumRRO) 

00  North  Washington  Street 
lexandria,  Virginia  22314 

10.  Project/Task/ Work  Unit  No. 

2Q062107A712 

11.  Contract/Grant  No. 

DAHC- 19-70-C-0012 

Sponsoring  Organization  Name  and  Address 

Iffice,  Chief  of  Research  and  Development 
lepartment  of  the  Army 

Washington,  D.C.  20310 

13.  Type  of  Report  &  Period 

Covered 

Research  Product 

14. 

.  Supplementary  Notes 

HumRRO  Division  No.  4,  Fort  Benning,  Georgia 

Technical  Advisory  Services  to  the  U.S.  Army  Infantry  Board 

,  Abstracts 

>This  document  presents  in  brief  and  readable  form  a  guide  for  the  development 

of  test  questions  for  questionnaire  instruments  other  than  achievement  testing. 
The  content  is  restricted  to  attitude  and  information  type  questions,  but  should 
help  the  novice  avoid  some  of  the  more  serious  errors  that  can  produce  faulty 
questions,  and  the  inevitable  consequence,  faulty  information. 


17.  Key  Words  and  Document  Analysis.  17a.  Descriptors 

‘Guides 

♦Questionnaire  Survey 


17b.  Idcntifiers/Open-F.nded  Terms 

Questionnaire  Development 
Training  Aid 


17c.  COSATI  Field/Group  05  02  Behavioral  and  Social  Sciences,  Information  Sciences 


18.  Availability  Statement 

19.  Security  Class  (This 

21.  No.  of  Pages 

Approved  for  public  release; 

RepnrUuj^ciAssiFIED 

27 

distribution  unlimited 

20.  Security  Class  (This 

PnKp)  UNCLASSIFIED 

22.  Price 

FORM  NTIS-39  (10-70)  USCOMM-DC  40329-P7  1 


Research  By-Product 


A  GUIDE  FOR  DEVELOPING 
QUESTIONNAIRE  ITEMS 

T.  0.  Jacobs 
January  1970 


Approved  for  public  release; 

distribution  unlimited 


Technical  Advisory  Service 


This  document  does  not  represent  official  opinion 
or  policy  of  the  department  of  the  Army. 


HumRRO  Division  No.  4 
Fort  Benning,  Georgia 


HUMAN  RESOURCES  RESEARCH  ORGANIZATION 


FOREWORD 


The  purpose  of  chis  document  Is  Co  present  in  brief  and  readable 
form  a  guide  for  Che  development  of  Cesc  questions  for  questionnaire 
instruments  other  than  achievement  testing.  The  guide  was  originally 
developed  as  a  part  of  Technical  Advisory  Service  provided  to  the 
U.S.  Army  Infantry  Board,  Fort  Bennlng,  Georgia.  It  is  published  here 
as  a  Research  By-Product  because  of  its  potential  utility  to  other 
organizations  and  agencies  in  developing  questionnaire  items. 

The  guide  was  developed  at  HumRRO  Division  No.  4,  Fort  Penning, 
Georgia.  Dr.  T.  0.  Jacobs  is  Director  of  the  Division.  Kill fury 
liaison  end  support  were  supplied  by  the  U.S.  Army  Ir.Iantry  Human 
Research  Unit  of  which  L1C  Chester  I.  Christie  is  the  Chief. 

HumRRO  research  for  the  Department  of  the  Army  is  conducted  under 
Contract  CAHC  19-70-C-0012.  Training,  Motivation,  Leadership  Research 
is  conducted  under  Army  Project  2Q062107A712 . 


Meredith  P.  Crawford 
President 

Human  Resources  Research  Organization 


CONTENTS 


FOREWORD  .  i 

PREPACE  .  iii 

INTRODUCTION  .  1 

CHAPTER 

1  Ask  Yourself  .  2 

Who  Needs  the  Information?  . .  2 

What  Decision  will  be  Made  Based  on  Your 

Information  .  2 

What  Facts  will  Affect  the  Decision?  .  3 

Whom  Are  You  Asking?  . 3 

What  Are  the  Consequences  of  a  Wrong  Answer .  4 

2  Question  Types  .  5 

Open-Ended  Questions  .  5 

Two-Way  Questions  .  6 

Multiple-Choice  Questions  .  7 

3  Framing  the  Questions  .  11 

The  Initial  Development  of  the  Question .  11 

Pretesting  the  Questionnaire  .  19 

4  A  Check  List  . . 21 

A.  The  Foundation  .  21 

B.  The  Question  Type  .  21 

C.  Writing  Questions  . 22 


li 


PREFACE 


The  purpose  of  this  document  Is  to  present  In  brief  and  readable 
form  a  guide  for  the  development  of  teat  questions  for  questionnaire 
Instruments  other  than  achievement  testing.  It  is  based  to  a  very  j 
great  extent  on  an  excellent  and  readable  book  by  Stanley  L.  Payne. 

The  book  by  Payne  is  a  great  deal  more  comprehensive  than  the 
present  document.  !Jhile  the  content  here  has  been  restricted  to 
attitude  and  Information  type  questions,  even  these  topics  are  not 
considered  in  the  technical  detail  that  would  be  apnropriate  for  a  test 
construction  specialist.  Further,  no  attempt  has  been  made  to  provide 
information  about  sample  selection,  sample  size  requirements,  counter¬ 
balancing  considerations  for  alternate  questionnaire  forms,  or  for 
control  of  possible  admlnistraticn/testing  variables. 

If  the  reader  can  obtain  access  to  the  book  on  which  the  present 
document  was  based,  he  will  find  it  readable,  and  thorough.  It  is 
recommended.  However,  in  its  absence,  the  present  guide  may  help  to 
avoid  at  least  a  few  of  the  more  serious  possible  errors  that  can  produce 
faulty  questions,  and  the  inevitable  consequence,  faulty  information. 


Payne,  Stanley  L.  The  Art  of  Asking  Questions. 
Princeton  University  Tress,  1951. 


Princeton: 


A  GUIDE  FOR  DEVELOPING  QUESTIONNAI  RE  ITEMS 


INTRODUCTION 

The  purpose  of  this  document  is  to  tAlk  about  how  to  ask  questions. 

This  is  not  as  easy  as  it  sounds.  The  object  is  to  ask  the  right  ques¬ 
tion,  and  to  ask  it  in  such  a  way  that  you  will  find  out  what  you  need 
to  know. 

There  are  a  great  many  pitfalls  in  asking  questions.  Not  all  of  them 
will  be  considered  here.  Further,  the  kinds  of  questions  with  which  we 
will  be  concerned  will  ask  mainly  for  information  that  someone  else  has, 
or  for  how  he  feels  about  something.  These  are  answers  that  a  test  soldier 
might  have  after  participating  in  an  equipment  troop  test,  for  example, 
or  information  he  might  have  about  a  combat  operation.  There  are  rules 
for  how  best  to  ask  this  kind  of  question.  Many  of  these  rules  will  also 
apply  to  the  kind  of  test  that  occurs  after  a  course  of  instruction,  l.e., 
an  achievement  test,  but  no  attempt  will  be  made  to  cover  all  the  problems 
and  techniques  that  are  important  in  developing  that  kind  of  test. 

Payne*  reported  the  results  of  asking  a  sample  of  researchers  what 
they  saw  as  the  principal  problems  with  research  methods,  with  the  fol¬ 
lowing  results: 


Improperly  worded  questionnaires  74% 
Faultv  Interpretation  58 % 
Inadequacy  of  samples  52% 
Improper  statistical  methods  44% 


Presentation  of  results  without  supporting  data  41% 

So,  this  monograph  will  concentrate  on  what  seems  to  be  the  largest 
part  of  the  problem,  writing  good  questions.  This  is  not  the  total  solu¬ 
tion  to  all  the  criticisms  mentioned  above,  but,  as  three  experts  in  every 
four  would  indicate,  it  is  a  good  start  in  the  right  direction. 

Four  chapters  will  follow  this  introduction.  In  the  first,  there  will 
be  certain  kinds  of  questions  you  should  ask  yourself.  This  is  to  make 
sure  that  you  know  what  you  are  asking  about.  In  the  second  chapter, 
there  will  be  a  discussion  of  types  of  questions,  together  with  their 
relative  advantages  and  disadvantages.  Some  types  of  questions  can  do 
certain  kinds  of  things  well,  but  do  other  things  fairly  poorly. 

The  third  chapter  will  discuss  actual  steps  in  building  a  questionnaire, 
including  some  reasonably  veil  tested  do's  and  don'ts.  The  fourth  and 
final  chapter  will  contain  a  check  list  that  will  help  avoid  many  of  the 
problems  Involved  in  writing  good  questions. 


*Payne,  Stanley  L.  The  Art  of  Asking  Questions. 
Princeton  University  Press,  1951. 


Princeton: 


CHAPTER  1 


Auk  Yourself 

The  words,  "systems  approach,"  arc.  almost  like  a  sacred  cow  these 
days.  It  anything  Is  done  using  a  "systems  approach,"  it  can't  be  all 
bad.  Seriously,  there  Is.  some  merit  to  taking  a  systems  approach  to 
asking  questions.  Asking  yourself  the  following  fire  questions  will  lay 
the  foundation  for  a  far  more  valuable  questionnaire  then  you  could 
otherwise  have  produced. 

Who  Weeds  the  Information? 

The  reason  for  asking  this  question  Is  so  straightforward  that  It 
may  seem  trivial.  However,  It  Is  something  which  can  be  overlooked, 
and  on  which  a  lot  depends.  Organizations,  Including  the  Army,  have  a 
machinery  for  transmitting  communications  within  Its  own  structure.  The 
term  usually  applied  is  Chain  of  Command.  The  requirement  for  the  Informa¬ 
tion  you  are  going  to  collect  may  have  come  from  two  or  three  levels  above 
your  own,  and  something  may  have  been  lost  in  the  communication  process. 

At  any  rate,  there  are  certain  things  you  need  to  know,  and  knowledge 
of  who  needs  the  Information  will  provide  you  a  source  In  the  event  you 
find  you  cannot  answer  the  following  four  questions.  This  leads  to  the 
next  question. 

What  Decision  will  be  Made  Based  on  vour  Information? 

The  answer  to  this  one  will  tell  you  in  part  why  the  Information  Is 
needed,  and  will  lay  the  foundation  for  the  answers  to  the  next  question. 
This  is  where  the  systems  approach  really  starts  to  make  a  difference. 
Depending  on  what  decision  is  going  to  be  made,  some  kinds  of  Information 
will  make  a  difference,  and  other  kinds  will  not. 

Suppose,  for  example,  you  are  preparing  to  collect  Information  as  a 
part  of  a  test  comparing  a  new  item  of  equipment  with  an  old  standard 
Item.  The  nature  of  the  decision  to  be  made  is  clear  enough.  It  trill 
be  either  selection  of  the  new  equipment,  or  retention  of  the  old  with 
which  It  Is  being  compared.  The  basis  for  the  decision  will  usually  also 
be  clear,  from  the  small  development  requirement  (SDR)  or  qualitative 
materiel  requirement  (QMR)  which  led  to  the  development  of  the  item  being 
tested.  Analysis  of  the  QHR  will  identify  the  qualitative  requirements 
the  new  equipment  must  have,  and  will  give  you  the  start  you  need  to 
develop  your  questions. 

An  explicit  statement  of  the  pending  decision  will  not  usually  lead 
as  easily  to  a  good  statement  of  your  information  needs  in  other  areas. 
However,  the  statemi  it  of  information  needs  Is  no  less  Important.  Knowl¬ 
edge  of  the  decision  that  Is  being  considered  Is  an  essential  key  in  both 
cases  to  the  next  step. 
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What  Facta  will  Affect  the  Decision? 


At  you  can  see,  Che  art  of  asking  pood  questions  may  be  closely  re¬ 
lated  to  the  designing  of  good  tests.  In  order  to  test  new  equipment 
well,  it  is  necessary  to  expose  that  equipment  to  the  kind  of  environ¬ 
ment  and  use  for  which  it  has  been  designed.  This  includes  use— and 
sometimes  abuse — by  the  type  of  individual  who  will  use  it  operationally. 

However,  sometimes  the  Information  is  needed  for  something  other  than 
an  equipment  test.  When  this  la  so,  the  task  of  finding  out  what  facts 
'fill  have  a  significant  bearing  on  the  problem  is  not  always  a  straight¬ 
forward  task,  and  sometimes  will  be  downright  difficult.  Sometimes  a 
decision  maker  \#lll  know  the  decision  being  considered,  but  may  even  be 
unsure  himself  of  the  full  range  of  information  that  will  be  needed  to 
make  a  good  decision.  In  this  case,  learning  what  kind  of  facts  will 
help  is  part  of  the  problem.  If  this  happens,  you  will  find  that  consulta¬ 
tion  with  colleagues  will  be  a  real  help,  asking,  in  eeoence,  "What  kinds 
of  things  would  you  want  to  know  if  you  had  this  decinlon  to  make?"  It 
will  help  to  make  a  list  of  elements  of  Information  that  you  feel  will 
be  needed  as  you  go.  If  you  have  time,  you  may  even  want  to  have  this 
list  reviewed  before  you  start  developing  questions. 

Clearly,  the  real  question  is,  "What  information  will  make  a  signif¬ 
icant  difference  to  the  decision  maker?"  If  you  collect  unnecessary  in¬ 
formation,  your  time  will  have  been  put  to  no  good  use,  at  the  very  least. 
Beyond  this,  a  considerably  more  serious  possibility  is  that  the  decision 
may  get  made  on  the  basis  of  Irrelevant  information.  This  leads  us  to 
our  third  question. 

Whom  Are  You  Asking? 

To  get  good  information,  not  only  must  a  good  question  be  asked,  but 
also  it  must  be  asked  of  someone  who  has  the  answer.  This  ties  in  with 
the  design  of  tests,  which  expose  proper  test  subjects  to  the  proper  con¬ 
ditions  so  that  they  will  then  be  able  to  tell  you  the  kind  of  information 
you  want  to  know.  In  fact,  one  basic  objective  of  a  good  equipment  test 
is  to  qualify  test  troops  to  be  able  to  answer  the  questions  that  need  to 
be  asked.  It  is  also  Important  to  be  sure  that  the  right  kind  of  test 
subject  is  used.  < 

However,  sometimes  it  is  necessary  to  get  Information  without  the 
benefit  of  a  carefully  designed  test.  When  this  is  so,  it  is  especially 
important  to  be  sure  you  are  asking  the  right  person  for  the  information, 
in  terms  of  whether  he  is  qualified  to  answer  and  whether  he  has  the  in¬ 
formation  you  need,  if  qualified.  This  may  even  require,  in  some  cases, 
that  ycu  actually  seek  out  an  expert  to  be  certain  of  getting  good  answers. 

It  does  not  require  much  knowledge  of  asking  questions  to  know  that 
a  question  about  combat  operations  should  not  be  asked  of  a  support  troop 
at  a  high-level  depot.  However,  sometimes  the  problem  is  more  subtle. 
Consider  the  following  question: 
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Example  I 

Was  the  TOE  of  your  unit  suitable 
for  the  missions  assigned? 

Yes  _ 

No  _ 


This  question  was  asked  of  noncommissioned  officers  In  Vietnam.  While 
It  appeared  to  be  a  perfectly  reasonable  question  to  the  officer  who 
wrote  it.  he  had  not  stopped  to  think  that  NCO's  generally  have  not  been 
trained  to  think  In  terms  of  how  TOE's  are  designed  to  produce  mission 
effectiveness.  It  is  quite  possible  that  many  noncommissioned  officers 
answered  this  question  on  the  basis  of  whether  they  thought  the  missions 
assigned  to  them  were  reasonable  missions,  which  Is  quite  a  different 
emphasis  than  that  requested  In  the  question. 

An  equally  Important  consideration  is  the  frame  of  reference  the 
test  subject  will  be  using  in  answering  the  question.  Frame  of  reference 
was  the  consideration  in  the  example  just  given,  in  which  the  NCOs'  frame 
of  reference  Is  simply  different  from  that  of  the  officer  who  worded  the 
question.  Frame  of  reference  is  a  topic  that  will  be  discussed  in  greater 
length  later. 

Another  topic  that  will  be  considered,  too,  is  whether  the  test  sub¬ 
ject  can  understand  what  you  want  in  a  question.  The  Army  has  an  expression 
that  Is  time  worn,  but  unbowed:  KISS.  This  stands  for  "Keep  It  simple, 
stupid!”  We  will  talk  more  about  this  later,  too. 

What  Are  the  Consequences  of  a  Wrong  Answer? 

While  this  basically  is  an  administrative  question,  it  has  an  im¬ 
portant  bearing  on  questionnaire  and  test  design.  Clearly,  if  it  makes 
little  difference  which  of  two  alternatives  are  chosen.  It  makes  little 
difference  if  the  information  is  collected.  On  the  other  hand,  if  there 
is  chance  that  millions  of  dollars  will  be  saved  through  the  use  of  a 
more  effective  block  of  training,  or  that  a  new  piece  of  equipment  might 
be  purchased  costing  millions  of  dollars,  but  which  is  not  better  than 
the  old,  it  is  necessary  to  design  tests  very  well,  and  ask  the  right 
questions  with  great  care.  Also,  in  general,  the  more  important  the 
test,  the  greater  will  be  the  number  of  test  subjects  needed  in  order  to 
provide  information  of  sufficiently  high  reliability. 

However,  these  are  questions  that  pertain  more  to  test  design,  and  the 
statistical  analysis  of  the  data  collected.  The  important  point  for  the 
present  discussion  is,  again,  that  when  you  ask  questions,  you  must  have 
a  feel  for  how  Important  it  is  that  the  answers  are  right. 
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CHAPTER  2 


Question  Types 

While  there  are  many  different  kinds  of  questions,  only  three  will  be 
considered  here,  because  it  will  almost  always  be  possible  for  you  to  get 
the  Information  you  need  with  one  or  more  of  them.  The  three  types  are 
open  end,  two-way,  and  multiple  choice.  In  deciding  which  type  you  want 
to  use,  you  will  need  to  consider  the  following  factors: 

1.  How  much  you  know  about  the  range  of  possible  answers. 

2.  How  much  time  your  respondents  have,  or  are  willing  to  give 
you. 

3.  The  number  of  subjects  required  to  get  satisfactory  reli¬ 
ability,  and  the  resulting  data  analysis  requirements. 

Open-Ended  Questions 

An  example  of  an  open-ended  question  is  shown  below. 


Example  II 

What  do  you  think  of  the  AR-15? 


One  advantage  of  the  open-ended  question  is  immediately  apparent. 

It  is  very  easy  to  ask.  This  is  useful  when  the  question  writer  either 
does  not  know,  or  is  not  certain  about  the  entire  range  of  possible 
alternative  answers.  By  using  this  type  of  question,  and  by  asking  enough 
respondents,  it  is  usually  possible  to  discover  almost  all  possibilities. 

However,  open-ended  questions  have  some  disadvantages,  too.  One  is 
that  it  usually  will  take  a  good  deal  more  time  for  you  to  get  answers, 
if  you  expect  the  answers  to  be  written.  Further,  depending  on  the  motiva¬ 
tion  of  your  respondent,  he  may  not  give  a  full  answer,  or  any  answer  at 
all,  when  he  sees  how  much  work  it  is  going  to  be  for  him.  A  second  dis¬ 
advantage  of  the  open-ended  question  is  the  requirement  for  analysis  time. 
For  a  question  asker  on  a  crash  project,  the  only  thing  that  could  be 
worse  than  failing  to  get  enough  Information  would  be  getting  so  much  that 
he  is  buried  by  it.  Open-ended  questions,  answered  by  motivated  respondents, 
are  good  at  burying  data  analysts.  They  usually  cannot  be  handled  by  machine 
analysis  methods,  e.g.,  computer  data  processing  methods,  without  lengthy 
preliminary  steps.  If  your  test  requires  a  fairly  large  number  of  sub¬ 
jects,  or  a  fairly  large  number  of  questions,  the  data  analysis  problem 
thus  can  grow  into  a  major  project  unless  some  other  form  of  question  is 
used.  Still  a  third  disadvantage  is  that  analysis  of  the  responses  to 
an  open-ended  question  usually  must  be  done  by  someone  who  has  substantial 
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knowledge  about  the  question's  content,  as  opposed  to  e  statistical  clerk. 
This  nay  result  In  Imposing  another  work  load  on  someone  who  Is  already 
hard  pressed  for  time. 

Finally,  a  fourth  disadvantage  Is  that  open-ended  questions  may  be 
easier  to  misinterpret,  because  the  respondent  do«»h  nut  have  a  set  of 
choices  available  which  might  in  themselves  guide  him  toward  the  proper 
frame  of  reference.  For  example,  the  following  question  Is  given  by 
Payne: 


. — — ~ — . . .  —  —  ■  ■  i 

Example  III 

Why  do  you  buy  the  50-cent  me  tor  oil 
Instead  of  the  70-cent  oil? 


A  fairly  obvious  reason  why  50-cent  oil  is  bought  instead  of  70-cent  oil 
is  that  it  is  cheaper.  However,  many  respondents  may  feel  this  is  so 
obvious  that  the  person  asking  the  question  wants  a  different  kind  of 
answer.  Thus,  the  question  may  fail  to  produce  valid  responses. 

In  summary,  the  open-ended  question  is  easy  to  ask,  but  it  has  several 
disadvantages,  among  which  is  that  It  produces  .data  that  are  time  con¬ 
suming  to  analyze.  Sometimes,  a  good  procedure  is  to  use  an  open-ended 
question  with  a  small  number  of  respondents  In  order  to  find  out  what  the 
range  of  alternatives  is.  Armed  with  this  knowledge,  it  may  then  be 
possible  to  construct  good  multiple-choice  questions  that  will  be  faster 
to  administer  and  easier  to  analyze. 

Two-Way  Questions 

The  two-way  question  Is  just  what  it  says,  a  question  which  permits 
the  choice  of  two  alternatives.  Example  IV  is  a  two-way  question. 


Example  tv 

Which  do  you  prefer,  the  ABC  helmet 
or  the  X/Z  helmet?  (Check  one) 

ABC  helmet  _ 

XYZ  helmet  _ 


An  advantage  Is  Immediately  apparent.  Wow,  your  respondent  can  merely 
check  one  and  proceed  to  the  next  question.  It  Is  a  relatively  easy  type 
of  question  to  develop,  and  permits  rapid  analysis. 
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However,  lc  also  has  disadvantages.  Plrst,  two  alternatives  night 
not  be  enough  for  some  types  of  questions.  Second,  there  may  be  a  tend¬ 
ency  for  your  respondents  to  choose  an  answer  based  on  what  Is  known  as 
a  response  set.  That  Is,  some  people  tend  to  choose  the  first  answei 
they  see,  while  others  may  choose  the  last  answer  they  see.  A  way  of 
avoiding  this  problem  with  the  two-way  question  is  to  print  half  the 
questionnaires  with  the  question  In  the  form  shown  in  Example  IV,  and 
the  other  half  with  the  order  of  mention  of  the  two  helmets  reversed. 
Sometimes  this  will  make  a  difference  and  sometimes  not,  depending  to 
a  substantial  extent  on  the  number  of  words  in  the  alternatives.  How¬ 
ever,  It  is  a  good  precaution  even  with  short  alternatives  like  those 
In  Example  IV. 

Another  common  fault  of  the  two-way  question  Is  the  presentation  of 
alternatives  that  overlap.  Example  V  has  overlapping  alternatives. 


Example  V 

Is  the  ABC  helmet  a  good  helmet, 
or  could  it  be  Improved? 

Good  helmet 


Could  be  Improved 


While  this  question  has  more  than  one  problem,  It  is  apparent 
that  It  Is  quite  possible  that  the  ABC  helmet  Is  both  a  good  helmet, 
and  could  be  improved.  In  addition,  almost  anything  can  be  Improved. 
Thus,  the  second  choice  ought  to  be  extremely  popular  if  your  respondents 
think  carefully  about  it,  without  regard  to  the  merits  of  the  helmet  as 
3uch. 

In  summary,  for  some  purposes  the  two-way  question  may  be  an  improve¬ 
ment  over  the  open-ended  question,  in  that  it  provides  for  faster  and 
more  economical  analysis  of  data,  which  means  that  data  can  be  gotten 
from  more  respondents.  However,  It  does  require  a  bit  more  care  in  its 
development.  Further,  it  is  also  subject  to  some  of  the  faults  of 
multiple-choice  questions,  as  well  as  some  peculiar  ones  of  its  own. 

Multiple-Choice  Questions 

This  type  of  question  has  the  advantage  of  being  easily  scored, 
which  means  that  data  analysis  is  a  relatively  inexpensive  process 
requiring  no  special  content  expertise.  It  also  requires  considerably 
less  time  per  respondent  answer  than  the  open-ended  question.  Fur  her, 
it  does  not  require  them  to  work  so  hard  to  give  you  the  information  you 
need,  which  means  they  are  more  likely  to  try  to  do  what  you  want. 
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Perhaps  an  even  mora  significant  advantage  is  that  It  puts  all  persons 
on  the  same  footing  when  answering.  That  is,  each  person  will  be  able  to 
consider  the  same  range  of  alternatives  when  choosing  hie  answer.  On  the 
other  hand,  this  may  be  a  major  disadvantage  for  the  question  maker,  be¬ 
cause  it  requires  him  to  know  all  the  significant  possible  alternatives 
at  the  time  he  formulates  the  question. 

One  of  the  big  problems  with  multiple-choice  questions  is  being  certain 
that  the  answer  you  receive  to  your  question  is  a  real  answer.  This  state¬ 
ment  may  sound  ridiculous,  but  the  fact  of  the  matter  is  that  a  very 
deliberate  and  well-thought-out  response  on  a  paper  and  pencil  multiple- 
choice  test  looks  very  similar  to  one  that  has  received  no  thought  at  all. 
That  is,  both  are  check  marks,  circles,  blackened  out  spaces,  etc.  How¬ 
ever,  there  are  a  few  ways  that  you  can  use  to  make  it  more  likely  that 
the  answers  you  get  are  well-thought-out. 

As  was  noted  in  discussing  the  two-way  question,  people  have  a  tendency 
when  choosing  answers  to  be  influenced' by' response  set."  With  multiple- 
choice  questions,  for  example,  they  tend  to  choose  answers  from  the  middle 
of  the  list,  if  the  list  consists  of  numbers,  and  from  either  the  top  of 
the  list  or  the  bottom  if  the  alternatives  are  fairly  lengthy  expressions 
of  ideas.  A  way  to  combat  this  is,  for  numbers,  to  use  different  forms 
of  the  questions  with  the  alternatives  in  a  different  order,  for  different 
ports  of  your  sample.  For  alternatives  presenting  ideas,  l.e.,  not  numbers, 
this  same  approach  can  be  used,  in  addition  to  keeping  the  number  of  words 
in  each  alternative  as  small  as  possible. 

Another  kind  of  error  is  the  inclusion  of  an  alternative  that  is  ob¬ 
viously  very  desirable.  Example  VI  contains  an  example  of  a  socially  un¬ 
desirable  response. 


Example  VI 

What  percentage  of  the  time  were  the  enemy 

targets : 

1 

a. 

Exactly  located  and  fired  at  _ 

_X 

b. 

Closely  located  and  fired  at  _ 

_X 

c. 

Location  unknown  but  fired  at 

hoping  to  suppreas  the  enemy  _ 

_X 

Alternative  c.,  if  the  respondent  thinks  carefully  about  it,  might  be 
thought  to  imply  a  lover  degree  of  skill  in  locating  the  enemy  than  might 
be  desirable.  In  thinking  about  the  item,  he  might  see  that  he  could 
check  b  Instead,  and  still  live  with  his  conscience.  Admittedly,  he  did 
not  see  the  enemy  but  he  can  be  fairly  well  convinced  that  the  enemy  was 
"closely  located,"  i.e.,  he  was  not  firing  purely  random  shots  as  alter¬ 
native  c  might  imply.  Thus,  the  pure  social  desirability  of  the 
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alternatives  ought  to  result  in  higher  percentages  reported  for  b, 
and  lower  percentages  for  c. 

Example  VIZ  shows  another  form  of  this  same  problem. 


Example  VII 

What  do  you  consider  the  most  Important 
characteristic  of  a  good  helmet? 

a. 

Comfort 

b. 

Stability 

c. 

Utility  for  washbasin 

d. 

Protection 

e. 

Weight 

If  the  rifleman  Is  asked  this  question,  he  almost  certainly  will  choose 
protection  as  Its  most  Important  characteristic.  However,  If  the  test 
helmet  is  being  procured  under  a  requirement  that  Its  protection  be  the 
equivalent  of  the  standard  helmet,  the  fact  that  protection  la  desired 
by  the  rifleman  is  Irrelevant,  because  the  test  helmet  will  fall  anyway 
If  Its  protection  capabilities  are  not  adequate.  This  gets  back  to  the 
requirement  for  a  systems  approach  to  the  asking  of  questions,  and  to  the 
"so  what?"  you  must  ask  about  your  own  questions.  In  this  particular  case 
the  question  would  have  had  a  nothing  'so  what."  Either  this  question 
should  not  be  asked,  or  It  should  be  asked  in  a  way  that  will  give  in¬ 
formation  about  what  the  soldier  considers  of  next  Importance  after  protec 
tion.  This  can  be  done  by  either  leaving  out  "protection,"  or  by  asking 
respondents  to  rank  the  alternatives. 

In  summary,  multiple-choice  tests  are  easy  to  administer,  and  fast 
to  score.  They  are  especially  useful  for  dealing  with  a  large  number 
of  test  subjects.  However,  the  questions  and  their  alternatives  must 
be  worded  with  very  great  care.  Otherwise,  the  information  obtained  from 
them  may  not  be  valid.  A  particular  requirement  for  multiple-choice  items 
is  that  the  question  writer  must  know  the  full  range  of  significant 
alternative.*.  He  can  help  himself  in  that  respect  by  asking  a  few  sub¬ 
jects  to  respond  to  open-ended  questions  In  advance,  however,  and  can  give 
himself  some  Insurance  by  providing  an  open-ended  alternative  which  can 
be  handled  in  the  data  analysis  through  special  processing,  as  is  shown 
in  Example  VIII. 
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Example  VIII 

What  types  of  missions  have  you  been  on? 

a.  Airmobile  assaults  _ 

b.  Mechanized  operations  _ 

c.  Cordon  and  search  _ 

d.  Base  security  _ 

e.  Other  _ _ 

(Specify  what) 
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CHAPTER  3 


Framing  the  Questions 

The  general  considerations  given  in  the  preceding  two  chapters 
should  help  In  giving  you  an  initial  idea  about  the  extent  of  the 
questions  you  want  to  ask  and  the  way  you  want  to  ask  them.  The  question 
of  what  infoimation  is  needed  should  have  been  answered  by  the  systems 
examination  of  the  requirement.  This  should  have  provided  a  compre¬ 
hensive  list  of  all  the  points  on  which  information  Is  required  for  the 
decision  that  Is  going  to  be  made.  The  brief  consideration  of  question 
types  should  have  given  you  an  Idea  of  how  you  want  to  ask  you^r  questions. 

From  this  point  on,  the  development  of  the  questionnaire  consists  of 
Initial  framing  and  rovlew  of  questions,  and  pretesting. 

The  Initial  Development  of  the  Question 

Let  us  assume  now  that  you  have  selected  a  particular  issue  or  element 
of  information  about  which  you  nov;  are  going  to  frame  a  question.  Here 
are  some  cautions  that  can  help  you  In  writing  questions: 

1.  Don't  assume  your  respondent  will  know  what  you  are  talking 
about,  lust  because  you  do.  Example  IX,  taken  from  Payne,  Illustrates 
this  point. 


Example  IX 

Which  do  you  prefer,  dichotomous 
or  open  questions? 


The  odds  are  that  a  fairly  substantial  number  of  people  would  not  be 
able  to  define  these  two  question  types  for  you.  However,  If  you  ask 
them  the  question  In  Example  IX,  they  will  be  happy  to  choose,  ine  point 
Is  that  people  will  not  volunteer  their  Ignorance  of  something,  though 
they  may  admit  It  If  you  ask  them.  However,  this  caution  goes  beyond 
ignorance  of  an  Issue.  Another  problem  is  that  the  specialist  wording 
the  question  may  simply  have  an  unusual  conmand  of  his  own  language. 
Scientific  jargon  has  been  criticized.  Perhaps  overlooked  Is  the  fact 
that  there  are  other  kinds  of  jargon,  too.  The  question  asker  has  a 
responsibility  to  make  himself  understood.  (One  way  of  screening  for 
Individuals  who  do  not  have  a  basis  for  providing  the  Information  you  need 
Is  to  Include  one  or  two  pure  Information  questions,  planning  to  discard 
questionnaire  returns  from  respondents  who  cannot  answer  the  information 
questions  correctly.) 
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2 .  Don’t  leave  yourself  wondering:  What  did  he  mean  by  that? 
Sometimes,  a  question  can  be  worded  ao  that  It  is  Impossible  to  know 
what  was  meant  by  a  given  answer.  Example  X  illustrates  this. 


Example  X 

Should  this 

cap  be  adopted,  or  its  alternate? 

Yes 

No  _ 

— 

If  he  answers,  'Yes,"  what  would  you  say  he  meant?  Of  course,  you  would 
rarely  find  a  question  with  such  obviously  ambiguous  response  alternatives, 
but  it  does  illustrate  the  problem. 

3*  Don’t  ask  people  to  go  against  their  basic  inclinations. 

This  is  a  very  vague  statement.  An  example  of  this,  however,  is  that 
people  generally  are  reluctant  to  criticize,  chough  they  enjoy  giving 
praise.  (There  are  some  exceptions  to  this  rule,  but  it  is  generally 
true.)  Thus,  a  question  that  allows  a  respondent  to  avoid  criticism  will 
bias  his  answers;  similarly,  a  question  that  leads  him  to  criticize  will 
bias  responses  because  he  will  not  wish  to  do  so.  Example  XI  illustrates 
this. 


Example  XI 

Was  your  unit's  use  of  the  following  correct, 
and  in  accordance  with  current  Army  doctrine? 

a.  Fire  and  maneuver 

Yes  _ 

Ho  _ 

If  no,  why  not?  _ 


This  question  asks  the  respondent  either  to  criticize  his  unit  or  to  avoid 
criticism.  Some  hardy  souls  might  answer  No,  if  they  have  an  important 
polut  to  make,  feeling  that  the  explanation  can  put  them  back  on  good 
footing.  However,  a  substantial  number  of  others  will  wash  their  hands 
of  the  whole  affair  and  answer  Yes. 
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4.  Don't  "lead.’1  You  hive  probably  hoard,  several  tinea,  the 
expression,  "People  are  no  d— ■  good!"  The  problem  with  this  is  that 
It  simply  is  not  true.  People  generally  are  reasonably  cooperative  and 
like  to  help.  If  they  can  figure  out  what  you  want,  they  will  try  to 
give  it  to  you.  Example  XII  contains  two  illustrations  of  leading,  both 
of  which  were  actually  used  in  the  collection  of  data. 


Example  XII 

Do  you  think  your  men  were  pretty 
highly  motivated  on  this  operation? 

Yes  _ 

No  _ 

Were  they  pretty  good  at  using  good 
noise  discipline  during  movement? 

Yes  _ 

No  _ 


As  one  might  predict,  the  impression  one  received  from  analysis  of  the 
answers  is  that  men  are  in  general,  highly  motivated,  and  use  good  noise 
discipline  during  movement.  (These  questions  also  suffer  from  allowing 
respondents  to  avoid  criticizing  the  problem  just  discussed,  and  of 
allowing  them  to  make  socially  desirable  answers  as  well.  This  is  a 
problem  that  will  be  discussed  later.) 

5.  Don't  confuse  or  lose  the  respondent.  The  language  you 
use  in  framing  your  question  will  have  a  lot  to  do  not  only  with  the 
quality  of  the  information  your  respondent  gives  you,  but  also  how  he 
feels  about  giving  it.  One  of  the  easiest  ways  to  create  a  problem  is 
to  use  a  confusing  question  such  as  that  given  in  Example  XIII  (taken 
from  Payne) : 


Example  XIII 

Are  you  against  not  having  prohibition  on 
non-week  days,  Including  Sunday  and  holidays? 

Yes  _ 

No  _ 
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This  particularly  Interesting  Illustration  might  mean  anything.  However, 
it  Is  not  necessary  to  use  double  negatives  to  confuse  someone.  It  can 
be  done  more  simply,  merely  by  using  words  he  cannot  understand,  as 
shown  In  Example  XIV: 


Example  ‘.IV 

What  countermeasures  were  taken  by  your  unit 
to  preclude  being  ambushed? 

a.  On  motor  movements  _ 


b.  On  foot  movements 


Of  course,  this  question  has  additional  problems  other  than  the  fact 
that  "countermeasures"  and  "preclude"  are  words  that  might  not  be 
understood  by  the  average  soldier.  An  additional  problem  ie  that 
a  respondent  who  knows  the  meaning  of  all  the  words  may  not  be  sure 
whether  he  should  give  countermeasures  that  preclude  ambush,  or  measures 
to  counteraihbushes.  There  Is  a  difference,  and  his  unit  might  have 
stressed  the  second. 

Example  XV  shows  how  a  respondent  can  be  asked  for  in¬ 
formation  he  may  not  be  able  to  give. 


Example  XV 

What  special  tactics  did  your  unit  employ  in 
conducting  the  following  types  of  operations? 
(Only  Indicate  actions  which  to  your  knowledge 
are  not  now  covered  In  FH's  or  other  published 
guidance . ) 

a.  Offensive  operations  (day,  night, 
mechanized,  river,  canal  line, 
raids,  ambushes,  patrols,  etc.) 


The  particular  source  of  trouble,  so  far  as  this  point  is  concerned, 
comes  from  the  section  In  parentheses.  While  the  Intention  of  the 
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question  author  1b  fairly  straightforward,  ha  has  virtually  ellmlnatad 
most  respondents,  bacausa  moat,  unlike  tha  author  of  tha  question,  are 
probably  not  all  that  familiar  with  "FM's  or  other  published  guidance." 

Of  course,  two  other  problems  should  be  mentioned  about 
this  particular  Item.  First,  the  word  "actions"  Is  ambiguous.  It  has 
two  meanings,  and  It  Is  difficult  to  know  which  one  is  correct  In  rela¬ 
tion  to  the  sentence  preceding  it.  Second,  this  Is  an  open-ended  question 
with  a  vengeance.  Given  a  motivated  respondent  in  a  highly  Inventive  unit, 
It  Is  possible  that  an  answer  to  this  single  question  could  take  hours. 

It  clearly  la  a  question  that  maximizes  the  convenience  of  the  writer, 
at  the  expense  of  the  respondent. 

Finally,  Example  XVI  shows  that  several  different  methods 
of  confusing  the  respondent  can  be  combined  in  the  same  question. 


Example  XVI 

When  engaging  enemy  targets,  seen  or  suspected,  at  ranges 

from 

0  to  50  meters  and  under  extreme  time  pressure,  what 

type 

fire  would  you  use? 

203 

a. 

Semiautomatic _ ;  automatic 

205 

b. 

Carefully  aimed _ ;  quickly  aimed  using  the 

200 

c. 

sights  _ ;  pointing  type  not  using  sights  fired 

from  the  shoulder _ ;  pointing  type  underarm _ . 

Single  rounds  ;  2-3  round  burst  ;  3-5  round 

178 

d. 

bursts _ ;  5-10  round  bursts  _ ;  20  round  burst 

hose  effect _ . 

Fire  at  single  targets _ ;  fire  at  groups  of 

186 

e. 

personnel _ ;  fire  at  the  area _ . 

Shoot  first  then  take  cover _ ;  take  cover  then 

shoot  _ ;  wait  for  orders _ . 

The  first  problem  Is  In  the  wording  of  the  initial  question.  It  is 
doubtful  that  the  average  Infantry  soldier,  when  reading  this,  would 
really  have  grasped  all  the  conditions  stated.  This  confusion  might 
be  reduced  by  rewording,  perhaps  in  this  way: 

What  type  of  fire  would  you  use  when  engaging 
very  close  targets  under  great  time  pressure? 

The  plan  here  is  to  make  the  question  more  understandable,  and  also  to 
break  the  very  complex  Initial  question  into  parts  that  can  be  handled 
more  easily. 


A  second  fault  with  Example  XVI  is  that  it  is  not  really 
clear  that  he  is  to  respond  to  each  line.  In  this  example,  numbers  are 
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given  at  tha  late  of  each  line,  allowing  how  many  answers,  iu  all,  ware 
given  to  all  tha  poaalbla  choice*  on  that  line  whan  thle  question  waa 
actually  uaad.  It  la  clear  that  while  almost  everyone  grasped  tha  idea 
that  ha  should  answer  more  than  one  time,  not  everyone  answered  the  same 
number  of  times.  However,  it  is  not  certain  whether  some  answered  too 
many  or  some  too  few.  Difficulties  like  this  can  be  avoided  by  giving 
the  respondent  explicit  directions,  and  by  making  his  job  more  simple 
than  thle  question  did. 

6.  Don't  make  vour  questions  too  long.  This,  and  the  pre¬ 
ceding  point,  are  similar  kinds  of  things.  The  more  words  a  question  has, 
and  the  bigger  they  are,  the  more  confusing  it  is  likely  to  be. 

Example  XVII  illustrates  this  problem. 


Example  XVII 

In  the  highly  specialized  counterinsurgency  environment 
represented  by  the  basically  internecine  affair  in 
Vietnam,  what  would  you  say  should  represent  the  basic 
essence  of  our  rationale  for  continuation  of  our 
Involvement? 

a.  Prolongation  of  attrition  of  enemy  forces,  in 
order  to  reduce  the  level  of  threat  to  South 
Vietnam. 

b.  Orderly  transfer  of  military  responsibility  to 
the  host  country,  in  order  to  produce  stabilized 
competency  to  deal  with  any  future  Internal 
disturbances. 


Several  things  are  wrong  with  this  question.  However,  it  is  unlikely 
that  the  average  respondent  would  be  bothered  by  them.  The  massive 
words  alone  very  probably  would  convince  him  to  stop  reading  at  an  early 
point.  He  might  then  choose  on  alternative  by  flipping  an  unbiased  coin; 
an  even  more  attractive  choice  might  be  just  to  go  to  the  next  item. 

Horst  of  all,  he  probably  will  choose  one  alternative,  but  not  in  an 
unbiased  manner,  so  that  your  item  not  only  will  fall  to  obtain  valid 
information  from  him,  but  will  obtain  invalid  information. 

The  four  "word"  rules  below,  adopted  from  Payne,  are 
very  good  to  follow: 

1.  Does  it  mean  wh3t  you  Intended? 

2.  Does  it  have  any  other  meanings? 

3.  If  so,  does  the  context  make  the  intended  meaning 
dear? 
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4.  Is  a  simpler  word  or  phrase  suggested  in  the 
dictionary? 

Perhaps  a  fifth  might  be: 

5.  Can  you  pet  along  without  It? 

The  point  of  this  caution  is  that  it  is  very  easy  to  get 
incorrect  information  on  a  multiple-choice  test  by  giving  your  respondent 
a  choice  he  cannot  understand.  Most  people  will  cooperate.  They  will 
choose  anyway. 

When  actually  writing  questions ,  each  should  be  carefully 
screened  for  words  that  might  be  unfamiliar  to  the  respondents  you  intend 
to  use.  Pretesting,  which  will  be  discussed  later,  can  also  provide  an 
opportunity  to  look  for  difficult  words.  A  good  technique  for  pretesting 
is  to  have  the  respondent  read  the  question  aloud  to  you  and  then  tell 
you  what  it  means.  Any  difficulties  at  all  in  the  pretest  should  cause 
danger  signals  to  fly. 

7.  Don’t  >jse  "All  American1  or  giveaway  words.  This  caution 
is  related  to  the  one  lust  discussed.  Example  XVIII  shows  the  use  of 
two  dead  giveaway  words  which  would  lead  the  careful  thinker  to  respond 
in  the  negative,  and  others,  thinking  less  carefully  to  respond  in  the 
positive. 


Example  XVIII 

Do  you  feel  that  your  unit  did  its  best 
in  all  contacts  over  the  past  six  months? 

Yes  _ 

No  _ 


One  wonders  if  any  unit  can  do  its  actual  best,  except  very  rarely. 

The  word  all  makes  this  an  even  more  difficult  question  to  answer 
positively.  You  probably  have  heard  the  expression:  All  statements 
containing  "all"  are  false,  including  this  one. 

The  result  of  a  question  like  that  shown  in  Example  XVIII 
is  that  it  may  separate  one  kind  of  thinker  from  another,  without  regard 
to  the  kind  of  information  you  are  trying  to  get. 

8.  Don't  load  your  questions.  Loading  questions  is  a  good 
deal  like  loading  dice.  Either  way,  you  can  be  pretty  sure  in  advance 
what  kind  of  result  you  are  going  to  get.  There  are  many  different 
ways  of  loading  questions.  Of  those  given  by  Payne,  the  following 
appear  to  be  the  most  likely  pitfalls  for  present  purposes. 
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a.  Ths  use  of  stereotypes.  If  motherhood  and  sin  were 
on  a  ballot,  ooaC  paople  would  vota  for  mo tharhood  and  against  sin. 
Example  XIX  shows  a  motharhood  type  of  question. 


Example  XIX 

Did  you  clean  your  weapon 
regularly  In  Vietnam? 

Yes  _ 

No  _ 


It  is  hard  to  imagine  anyone  answering  "No"  to  this  question.  As  you 
can  see,  this  is  very  much  like  leading,  also. 

b.  Recourse  to  the  status  quo.  People  have  a  substantial 
tendency  to  keep  things  the  way  they  are,  unless  something  Is  really 
wrong.  Example  XX  shows  a  question  that  should  have  been  worded  a  dif¬ 
ferent  way  to  avoid  this  problem. 


- 1 

Example  XX 

Do  you  feel  that  the  end-of-course  test  is 
satisfactory  as  it  Is,  or  should  It  be  changed? 

a.  Satisfactory  as  is  _ 

b.  Should  be  changed  _ 


Answers  to  this  question  should  be  suspected  merely  on  the  basis  of  the 
tendency  people  have  to  keep  things  the  way  they  are. 

c.  Unbalanced  alternatives.  Example  XXI  shows  loading 
of  a  different  type.  In  the  first  part  of  this  Example,  analysis  of  the 
available  alternatives  leaves  the  impression  that  the  writer  of  the  ques¬ 
tion  thinks  at  least  some  should  not  have  a  full  automatic  selector. 
Analysis  of  the  alternatives  in  the  second  part  of  the  Example  leads  to 
the  certain  conclusion  that  the  writer  of  the  question  believes  there 
should  be  at  least  one  grenade  launcher  in  the  rifle  squad. 
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Example  XXI 

Should  every  rifleman  In  the  rifle  equad  have 
a  full  automatic  aelector  on  hla  rifle? 

Yea  _ 

No  _ 

If  no,  how  many  should?  _ 

How  many  grenade  launchers  (M79)  do  you  desire 
In  the  rifle  squad? 

a.  1 

b.  2 

c.  3 

d.  4  or  more 


This  leads  to  the  observation  that  any  blew  on  the 
part  of  the  question  writer  himself  lust  might  be  one  source  of  load¬ 
ing.  It  turns  out  that  this  is  often  very  true,  and  most  of  the  time 
is  not  recognised  by  him  at  all.  So  a  good  check  here  is  to  ask  your¬ 
self  what  you  think,  what  someone  who  disagrees  with  you  would  think, 
and  whether  your  alternatives  would  give  him  a  good  chance  to  present 
his  views. 

Pretesting  the  Questionnaire 

Even  the  most  careful  screening  will  not  find  every  problem  with  all 
questions.  Pretesting  thus  is  very  important,  to  find  those  overlooked 
problems  that  would  otherwise  reduce  the  validity  of  the  information 
you  will  get.  However,  Just  any  pretest  will  not  do.  You  must  know 
how  to  pretest  the  items,  and  what  to  look  for. 

Perhaps  of  first  importance,  the  pretest  respondents  should  be  repre¬ 
sentative  of  your  eventual  target  audience.  That  is,  if  a  test  is  going 
to  be  run  using  TOE  Infantry  soldiers,  you  would  not  want  to  pretest  your 
questionnaire  with  a  sample  drawn  from  the  Infantry  Officer  Advanced 
Course.  If  your  eventual  test  sample  will  consist  of  Infantry  AIT  gradu¬ 
ates,  you  probably  would  not  want  to  pretest  with  School  Support  Troops, 
many  of  whom,  at  the  time  of  the  present  writing,  have  had  combat  ex¬ 
perience. 

A  second  point  is  that  the  pretest  is  more  useful  if  it  is  conducted 
by  someone  who  knows  the  subject  area.  If  the  question  writer  himself, 
for  example,  conducts  the  pretest,  using  pretest  respondents  one  at  a 
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time,  he  can  often  obtain  useful  Information  by  asking  for  the  ques¬ 
tion  to  be  read  and  then  explained.  Misunderstandings  become  very 
apparent  through  this  process. 

Another  useful  technique  Is  to  ask  the  respondent  to  explain  what 
he  means  by  a  choice,  or  to  give  his  reason  for  the  choice,  even  though 
these  questions  will  be  not  asked  in  the  main  administration  of  the 
questionnaire.  These  questions  will  frequently  reveal  incorrect  assump¬ 
tions  and  possible  rationales  that  the  question  writer  never  dreamed 
possible. 

While  pretesting  the  questionnaire,  a  high  proportion  of  respondents 
giving  no  response  of  a  "Don't  know"  response  should  raise  danger  flags. 
However,  especially  for  a  multiple  choice  question,  a  low  number  of 
"don't  know"  responses  does  net  guarantee  that  the  question  Is  good. 

By  recourse  to  the  methods  mentioned  in  the  preceding  paragraph,  the 
pretester  can  come  closer  to  guaranteeing  that  the  questions  are  good. 

Pretesting  may  seem  a  wa3te  of  time,  especially  when  the  author 
may  have  asked  several  people  In  his  own  office  to  critique  the  ques¬ 
tions,  or  perhaps  even  has  asked  a  questionnaire  specialist  to  critique 
it.  However,  it  will  usually  be  an  investment  that  is  well  worthwhile. 
It  is  crucial  if  the  decision  that  will  result  from  the  information  is 
Important . 


CHAPTER  4 


A  Check  List 

In  the  preceding  chapters,  we  have  talked  first  about  the  kind  of 
Information  you  need  to  establish  as  a  foundation  before  you  start 
writing  questions,  about ’the  types  of  questions  among  which  you  may 
choose,  and  about  the  kinds  of  problems  you  may  confront  as  you  develop 
the  questions  for  your  questionnaire.  The  following  check  list  may  be 
of  use  to  you  both  as  a  review  and  as  a  way  of  refreshing  your  memory 
as  you  actually  work  through  the  steps  of  developing  a  questionnaire 
Instrument. 

A.  The  Foundation. 

1.  Learn  who  needs  the  Information. 

2.  Learn  In  specific  terms  what  decision  is  going  to  be  made 

based  on  your  information. 

3.  Outline  the  facts  that  will  be  needed  to  make  those  decisions. 

4.  Identify  respondents  who  have  the  information  you  need. 

5.  Use  a  procedure  that  will  provide  Information  of  sufficient 

reliability. 

B.  The  Question  Type. 

1.  Open-ended  questions. 

a.  Be  sure,  before  asking  an  open-ended  question,  that  it 
cannot  be  replaced  with  a  multiple  choice  question. 
(Sometimes  it  cannot.) 

b.  Before  using  an  open-ended  question,  be  certain  that  you 
x;lll  have  enough  time  to  conduct  an  analysis  of  the  data. 

c.  Hake  a  particularly  strong  effort  to  avoid  open-ended 
questions  when  the  respondent  does  not  have  education 
beyond  the  eighth-grade  level.  (Some  would  have  said 
high  school  level.) 

2.  Two-way  questions. 

a.  Be  certain  that  two  alternatives  are  enough. 

b.  Try  to  use  two  forms  of  the  question,  in  Lvo  different 
questionnaires,  with  the  order  of  appearance  of  the 
alternatives  reversed  from  one  to  the  other. 
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c.  Be  cercaln  Chet  the  alternatives  do  not  overlap. 

d.  Be  certain  you  will  know  what  "was  really  meant"  by  each 
anawer. 

3.  Multiple-choice  questions. 

"a.*  Check  to  be  certain  “that  you  have  thoughtf'of'  all 

reasonable  alternatives.  (Cross  check  this  during  pretesting.) 

b.  If  you  are  asking  people  to  choose  from  a  list  of  numbers, 
be  certain  that  "response  set"  tendencies  to  choose  from 
the  middle  of  the  list  will  not  bias  responses  to  your 
question. 

c.  Similarly,  If  you  have  alternatives  that  are  lengthy,  make 
sure  the  tendency  to  choose  from  the  beginning  or  end  of 
the  list  does  not  bias  your  answers. 

d.  Be  certain  the  item  does  not  include  an  alternative  that 
will  "wash  out"  the  other  alternatives. 

Writing  Questions. 

1.  Initial  development. 

a.  Don't  use  terms  or  ask  questions  about  topics  that  your 
respondent  will  not  know  about. 

b.  Make  sure  that  any  answer  the  respondent  makes  will  have 
a  clear  meaning  to  you. 

c.  Don't  ask  people  to  make  undesirable  choices. 

d.  Don't  lead  your  respondents  by  wording  the  item  so  they 
will  know  what  you  want. 

e.  Make  the  item  and  the  choices  short,  and  use  simple  words. 

f.  Make  certain  the  item  clearly  specifies  what  you  want  the 
respondent  to  do. 

g.  Don't  ask  him  to  do  several  things  in  one  item. 

h.  Don't  ask  for  several  kinds  of  information  in  one  item. 

1.  Don't  use  tricky  wording,  especially  double  negatives. 

j.  Don't  ask  for  fine  distinctions,  unless  you  know  they 
will  be  meaningful  to  your  respondents. 
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k.  Don't  word  your  question  in  ouch  a  way  that  the  balance  of 
reaponaea  will  unavoidably  be  in  one  direction  (loading). 

l.  Be  aura  your  anavera  are  worded  ao  that  aoneona  who  dlaagreea 
with  you  can  find  an  anawer  preaantlng  hla  aide  of  the  iuaue. 

m*.  Screen  your  items  'for  "All  American"  words  or  dead*glveaway 
worda. 

2.  Pretesting. 

a.  For  pretesting,  uae  subjects  representative  of  your 

*  eventual  target . 

•  b.  Ask  your  respondent  to  provide  both  an  answer  and  hla  reason 

for  the  answer. 

c.  Ask  If  there  are  other  answers  he  could  have  given. 

3.  One  last  look. 

a.  Re-read  each  question,  and  apply  the  "so  what"  criterion. 
(Will  this  Information  really  make  a  difference  to  the 
decision  maker?) 

b.  Re-examine  each  of  the  elements  of  Information  originally 
Identified  as  necessary,  and  be  sure  that  there  is  at  least 
one  question  on  each,  and  that  the  responses  will  provide 

a  clear  basis  for  that  part  of  the  decision. 

c.  Read  the  questionnaire  as  a  whole  to  check  that: 

(1)  The  flow  of  items  as  a  whole  will  not  confuse  the 
respondent . 

(2)  The  flow  of  items  as  a  whole  will  not  tip  him  off  as 
to  what  answer  you  think  Is  correct. 

Using  this  check  list,  and  the  thoughts  contained  in  the  preceding 
chapters,  will  not  guarantee  a  successful  questionnaire  or  a  successful 
test.  However,  it  is  hoped  that  they  will  make  the  job  a  little  faster, 
or  a  little  easier,  and  a  little  better. 
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